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composing an image from the two inputted images based upon thfe above 
detemiined amount of the difference in focus. 



determination further comprising: 

iteratively low-pass filtering the common in-focus area of one of the inputted 
images; 

determining whether or not the low-pass filtered common in-focus area 
substantially matches the common in-focus area of another inputted images; and 

determining the amount of the focus difference in the one common in-focus area 
between the two images based upon a number of the iterative low-pass filtering. 



The Examiner rejected independent claims 1, 7 and 13 under 35 U.S.C. § 102(b) 
as being anticipated by Kodama et al (1966). On the other hand, the Examiner indicated 
the objection to claims 2 through 6, 8 through 12 and 14 through 16 as being dependent 
upon a rejected base claim, but would be allowable if rewritten in independent form 
including all of the limitations of the base claim and any intervening claims. 

Newly amended independent claims 1, 7 and 13 now each explicitly recite 
"determining a relative position between two of the images based upon a predetermined 
set of movements. . . As supported by the original disclosures at lines 7 through 18 on 
page 8 of the current application, the two images may be taken at a different focal length 
as well as with a different optical axis as long as the two images contain the same objects. 
The relative position of these images is determined "based upon a predetermined set of 
movements" as explicitly recited by newly amended independent claims 1, 7 and 13. 




area is user-adjustable. 
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In contrast to the above described feature, the cited reference discloses a set of 
formula that describes how the desired all-focused image is related to multiple images at 
a different focal length. The relationships are described in equations (1) through (6). 
Although it is not explicitly stipulated, the images are taken at a focal length but along 
the same optical axis since no image transformation is disclosed between the two images 
before the above equations are applied. 

Based upon the above quoted patentable feature, the Applicants respectfully 
submit to the Examiner that newly amended independent claims 1, 7 and 13 are not 
anticipated by the cited reference. Similarly, dependent claims 2 through 6, 8 through 12 
and 14 through 16 ultimately depend from the newly amended independent claims and 
incorporate the patentable feature. Thus, the Applicant respectfully submit that the 
rejections of claims 1 through 17 should be withdrawn. 



Newly Added Claims 

The Applicants have added claims 17 through 29 to the current application in the 
current response. Newly added independent claims 1 7, 22 and 27 have the combined 
subject matter limitations of independent claims 1, 7 and 13 as well as those of objected 
claims 2, 8 and 14. As the Examiner indicated the allowable subject matter limitations, 
the Applicants respectfully submit that newly added independent claims 22 and 27 
should be allowable. Since newly added dependent claims 18 through 21, 23 through 26, 
28 and 29 ultimately depend from newly added independent claims yf, 22 and 27 and 



incorporated the allowable subject matter limitations, newly added claims through 
should be patentable distinct. 

These claims are added without introducing new matter, and they have been 
supported by the original disclosures of the current application. The Applicants 
respectfully submit that the newly added claims should be entered and allowed. 
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In view of the above amendments and the foregoing remarks, Applicant 
respectfully submits that all of the pending claims are in condition for allowance and 
respectfully request a favorable Office Action so indicating. 



Respectfully submitted, 



Date: July 8, 2003 




Reg. No. 37,009 

KNOBLE & YOSHIDA LLC 
Eight Penn Center, Suite 1350 
1628 John F. Kennedy Blvd. 
Philadelphia, PA 19103 
(215) 599-0600 
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I(x) = f(x) + g(x) (2) 

then, 

5 I,(x) = f(x) + h2g(x) (3) 

l2(x) = h,f(x) + g(x) (4) 

where hi and h2 are blur functions. According to this model, equations (2) and (4) lead 
10 to 

G(x) = HI(x) (5) 

and 

15 

G(x) = (hr 1) I,(x) + (h2-l) l2(x) (6) 

thus, the following equation is obtained. 

2 0 H = hih2-1 (7) 

For each input image, if the blur functions hi and h2 are known, using the initially 
reconstructed image 10 = II or 12. 

2 5 . Ui = (H+l)In-G (8) 

By iteration, the universally focused image is obtained. 



30 



Similarly, an arbitrarily focused image lab is expressed as follows when blur 
functions ha and hb are respectively used for a near image f(x) and a far image g(x). 
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lab(x) = haf(x) + hbg(x) 



(9) 



Instead of Equation (8), 



5 



Gab = (hbhi-ha) Ii(x) + (hah2-hb) I2(X). 



.(10) 



is considered 
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U, = (H+1) In-Gab 



-(11) 



for the iterative process for generating the universally focused image. Furthermore, the 

Kodama et al. reference discloses two ways to determine the blurring function. The first 

one uses predetermined image capturing conditions including the focal point and at each 
15 distance, the blu rring function is measured for an object. The second way is that the 

blu rring function is estimated by adjusting the blumess of each pixel as described in the 

Naito et al. reference. 

The above described two groups of prior art techniques are susceptible to errors in 

generating a universally focused image. Li general, the conventional select and merge 
2 0 methods are subject to selection errors, which cause an inferior merged image. The errors 

are particularly likely near edges where the intensity changes are larger in an out-of-focus 

image portion than an in-focus image portion. 

For the prior-art iterative reconstruction methods, in general, substantially no 

selection errors are involved. The iterative reconstruction methods do not need 

2 5 information on the location of blumess, and it is possible to generate not only a 

universally focused image but also an arbitrarily focused image. On the other hand, it is 
not clear how many iterations are necessary to converge although the Kodama reference 
discloses only about three iterations. For a large number of iterations, an amount of 
calculation increases, and it takes a substantial amount of time. In addition, the two 

3 0 proposed techniques for determining the blurring functions also have the following 



4 



RCOH-1021/AP99-382 



PATENT 



problems. For the first measurement technique, it is necessary to measure the 
characteristics of the camera as well as other information on each input image such as the 
focal point length and exposure which affect the blurring function. For these reasons, 
images taken by an auto focus camera are not usually appropriate for the measurement 
5 technique since the auto focus camera generally does not provide information such as the 
focal point length. The other technique to determine the blurring function based upon the 
estimation from the image generally requires a large amount of calculation, and it takes a 
large amount of time. Furthermore, it is difficult for the user to adjust the blurring 
function. 

10 In addition to the above-described undesirable problems, the prior-art methods 

also have the following difficulties. Since the zoom ratios usually change as the focal 
point length is changed, it is necessary to correct at least the zoom ratio of each image 
before the portions of the images are merged into one composite image. For example, 
after an auto focus camera is positioned and captures a first image in which a 

15 predetermined object is centered, to prepare for capturing a second image, the camera is 
turned away from the predetermined object and the background scene is focused with the 
release button for example . Then, with the above described focus, the camera is turned 
back towards the predetermined object, and the second image is captured. The second 
image is centered around the predetermined object. These two images have not only 

2 0 different zoom ratios but also slightly different orientations and positions. The ratio 

correction alone does not allow the precise positioning of image portions. Images taken 
by a regular camera without a beam splitter and two CCD's require a high degree of 
flexible re-positioning technique which includes parallel and rotational movements. 

In order to solve the above-described problems, it is desired to quickly generate a 

2 5 high-quality pan focus composite image data from a plurality of common images captured 

by a generally available auto focus and auto exposure camera. To accomplish the above 
objectives, it is desired 1) to speed up the image composition process based upon the use 
of a blurring fiinction, 2) to speed up the blurring function determination process, 3) to 
facilitate the confirmation and correction of the results by the blurring function, 4) to 

3 0 speculate the relative position of image data and to enable the position match for the 
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the relative position is acceptable in act 106. When the result is to be corrected again, the 
preferred process goes back to the act 102. On the other hand, when the result is not to be 
corrected or accepted, the preferred process goes to act 108, where an in-focus area is 
determined. 

5 To determine the in-focus area, the act 108 relies upon the characteristic that the 

in-focus area has an increased amount of high-frequency region than the out-of-focus 
area. The in-focus area determined in act 108 is corrected in act 1 10. Then, the preferred 
process determines an input blurring function in act 1 12 and corrects the input blurring 
function in act 1 14 if desired. Similarly, the preferred process determines an output 

1 0 blurring function in act 1 16. The act 1 16 outputs an arbitrarily focused image by 
specifying an output result corresponding to (ha, hbb) in equation (9). This act is not 
necessary when a universally focused image is to be outputted. The act 1 16 is similar to 
the act 1 14 in that an in-focus input image and a series of images produced by multiplying 
the input image by a set of blurring functions. A user is requested to select one of the 

1 5 series of the images, and the blurring function (ha, hbb) used to generate the selected 

image is outputted. When an image without a blurring function is selected, a universally 
focused image is outputted. Based upon the above-determined data, the preferred process 
performs the composition of the images in act 1 18 before the composite image is 
outputted in act 120. 

2 0 Using the determined blurring functions (hi, h2, ha, hbb) from the acts 114 and 

116, the output of the composition process is an iterative reconstruction method. In 
general, the composition process is an iterative reconstruction method. Although the 
averaged pixel data after the two input images have been composed based upon the 
already determined relative position has some blur, the averaged pixel data is a good 

2 5 approximation of the universally focused image. Based upon the above advantageous 

feature, the average pixel data is used as an initial data set, and the equation (1 1) is 
reiterated only once in order to save a substantial amount of time. 

The initially reconstructed image is PO=(Il+I2)/2, and the equation (1 1) is 
performed once. After the above processing, the reconstructive image I^I is expressed as 

3 0 follows: 
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L= - + K y ^{^-KK^ K y (12) I 

The above processing multiples each image data Ii and I2 by a filter (hih2/2-hihb+ha), 
5 (hih2/2-h2ha+hb) which is determined by a blurring function (hih2,ha,hb) and adding the 
products. The above universally focused image or the arbitrarily focused image are 
outputted to output devices or storage such as a hard disk 210, a floppy disk 212 and a 
CRT display unit 204. 

Now referring to FIGURE 2, a block diagram illustrates one preferred 

10 embodiment of the image composition system according to the current invention. The 
preferred embodiment includes a central processing unit CPU 200, a memory unit 202 
where software or programs run and various input/output devices are connected to the 
CPU 200 via a bus 201. The input/output devices further include a cathode ray tube 
(CRT) display unit 204 with a display controller 203; input devices such as a keyboard 

1 5 207 and a mouse 208 with an input interface 206; memory storage media such as a hard 
disk 210, a floppy disk 212 with a floppy disk driver (FDD) 21 1 and a disk controller 209; 
and an image input unit 214 with an input interface 213. For example, the software for 
composing the image may be stored in a storage medium such as a floppy disk 212 and is 
read into the hard disk via the floppy disk drive 211. When the composition program is 

2 0 run, it is loaded into the memory 202. In addition, other storage media such as an optical 
medium with an optical drive alternatively store the composition program. To input an 
image, one way is to directly input a digital image captured by a digital camera via the 
input interface 213. Another example is to scan in via the input interface 213 an image 
printed from a silver halide film. Yet another example is to input an image from the 

2 5 floppy disk 212. 

Now referring to FIGURE 3, one exemplary software implementation shows a 
display screen 204 where the act 104 of the preferred process is performed. A first image 
301 and a second image 302 are overlappingly displayed. The first image 301 is a 
standard while the second image 302 is the result of the processing. The display screen 
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204 includes a message or instruction 303 to a user, an O.K. button 304 for accepting the 
displayed image and a recalculate button 305 for initiating the calculation process. The 
user observes and determines whether or not the two images 301 and 302 correctly 
overlap. If the user determines that the two images correctly overlap, he or she clicks the 
5 O.K. button 304 via the mouse 208 of FIGURE 2. On the other hand, when the user 
determines that the two images do not correctly overlap, he or she can either manually 
move one image over the other via the mouse 208 or clicks the recalculate button 305 for 
automatically re-calculating the correct position. In any case, the user repeats either a 
combination of the above-described options until he or she feels satisfied that the 

10 overlapping requirements is -are met. Upon satisfaction, he- the user clicks the O.K. button 
305. When the O.K. button 50§-304_is clicked, the relative position of the two images 
301 and 302 is determined, and the preferred process proceeds to the act 108 of the flow 
chart as shown in FIGURE 1 . On the other hand, when the recalculate button 30 ^305 is 
clicked, the relative position is recalculated in the act 102. 

15 Now referring to FIGURE 4, an exemplary high-pass filter to determine an in- 

focus area is illustrated. The image data is divided into blocks of 16 x 16 pixels. After 
processing with the high-pass filter having the coefficients as shown, each block is added. 
Assuming that Bij is the sum of pixel values in a block] from the image data i, (Blj - 
B2j) is larger as a high-frequency portion is larger in a first image 1 than second image 2. 

2 0 In other words, the first image 1 is brighter than the second image. The block having the 
highest value in Blj-B2j is considered as an in-focus image area of the first image 1. 
Similarly, the block j having the lowest value in Blj-B2j or having the highest value in 
B2j-Blj is considered as an in-focus image of the second image 2. 

FIGURE 5 shows an exemplary software implementation of the act 1 10 of the 

2 5 preferred process as shown in FIGURE 1 for accepting or correcting the above- 

determined in-focus image areas. The exemplary software implementation shows an 
image 501, which is either the first image or the second image and an in-focus area 502. 
The exemplary screen also includes the instruction 503 as well as an O.K. button 504. If 
the user determines that the in-focus area is correct, he or she clicks the O.K. button 504. 

3 0 On the other hand, if the displayed in-focus area is not acceptable, the user moves the box 
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502 to a correct in-focus area via the mouse 208 and clicks the O.K. button 504. The 
above-described operation is repeated for each of the two images. Based upon the above 
in-focus area as determined in the act 1 10, a blurring function is applied only to the in- 
focus area as described in the publication entitled "Enhanced Image Acquisition by Using 
5 Multiple Differently Focused Images" by Naito et al. The application of the blurring 
function produces gradually and sequentially blurred images so that the in-focus area 
gradually matches with a corresponding out-of-focus area. The gradual application 
determines the best matched blurring function of the out-of-focused area. 

Now referring to FIGURE 6, a low-pass filter is used to gradually and iteratively 

1 0 blur an image, and one example of the low-pass filter is illustrated by a set of low-pass 
filter coefficients. In order to determine a blu rring function for a second image data set 
with respect to a first image date -data set that has an in-focus area A, a corresponding out- 
of-focus area is designated as B in the second image data set. Using the above 
designations, the image data A after a first application of the low-pass filter is designated 

15 as A(l), and a second application to the image data A(l) is designated as A(2). Similarly, 
after n-1 applications to the image data set A is designated as A(n-l), and after nth 
applications, the image data A is designated as A(n). The sum of a difference between 
A(n) and the corresponding B in absolute value for each pixel is designated as D(n). 
According to the above-described procedure, D(l), D(2), D(3) ... D(n) are sequentially 

2 0 determined, and the minimal value is sought among the above sums, n is a predetermined 
number. D(l), D(2) and D(3) gradually decrease towards D(n). D(n) is considered to be 
the minimal when D(n)>D(n+l) is confirmed. Thus, the corresponding image A(n) is 
considered to be the closest to the image B. As the low-pass filter as shown in FIGURE 6 
is applied to an image for n times, a blurring function is determined for the second image. 

2 5 Similarly, a blurring function for the first image is also determined in the above- 

described manner. To determine a blurring function for the first image, an in-focus area 
of the second image is gradually blurred to match a corresponding out-of-focus area of the 
first image. 

Now referring to FIGURE 7, one exemplary software implementation shows a 

3 0 display screen where the act 1 14 of the preferred process is performed. The screen 
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